Skip to content

feat: opt-in dizzy VictoriaMetrics + Grafana stack and chaos targets for Keystone and Glance#718

Merged
berendt merged 9 commits into
mainfrom
implement/issue-717-dizzy-stack
Jul 23, 2026
Merged

feat: opt-in dizzy VictoriaMetrics + Grafana stack and chaos targets for Keystone and Glance#718
berendt merged 9 commits into
mainfrom
implement/issue-717-dizzy-stack

Conversation

@berendt

@berendt berendt commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in WITH_DIZZY observability stack to the kind dev stack and two
chaos make targets that run dizzy's
create/mutate/delete churn soak against a quick-start ControlPlane. dizzy exports
per-operation metrics over OTLP, which a VictoriaMetrics-single server ingests
natively at /opentelemetry/v1/metrics (no collector), fronted by an
anonymous-Viewer Grafana pre-provisioned with dizzy's three dashboards.

The stack is default-off: with WITH_DIZZY unset the overlay never renders, no
HTTPRoute attaches, and the new kind port mapping and Gateway listener sit idle
but ready, so the flag can be enabled later on any cluster created after this
change without recreating it.

Closes #717

Change set (in commit order)

The branch builds the stack bottom-up: host plumbing first, then the runner, the
overlay, the deploy-script gate, the make targets, and finally the Renovate,
test, and docs wiring.

  1. e2557c9c feat(kind): bridge host 8428 to VictoriaMetrics NodePort 30428
    Adds an unconditional extraPortMapping forwarding host 127.0.0.1:8428
    container NodePort 30428 — the host→cluster path dizzy (running on the
    developer host) uses to export OTLP metrics. Loopback-only; left in place
    regardless of the opt-in flag so the flag can be turned on later without
    recreating the cluster.

  2. 5009774d feat(kind): add https-dizzy Gateway listener and TLS certificate
    Adds an unconditional fourth HTTPS listener (https-dizzy) on the shared
    port 443, routed by SNI to dizzy.127-0-0-1.nip.io. This is the repo's first
    listener that admits routes from another namespace (keyed by the automatic
    kubernetes.io/metadata.name label), so the gated overlay's HTTPRoute
    attaches from the dizzy namespace without a ReferenceGrant. With
    WITH_DIZZY unset no route attaches and the SNI answers 404, mirroring how
    the glance listener exists without a Glance service. Ships a per-hostname
    self-signed Certificate matching the keystone/horizon/glance siblings.

  3. 6cdc2558 feat(hack): add dizzy.sh with dashboard staging and chaos runner
    New hack/dizzy.sh wrapping the dizzy tester. DIZZY_VERSION is the single
    pinned string (v0.1.0). stage-dashboards copies dizzy's three Grafana
    dashboards into the git-ignored deploy/kind/dizzy/dashboards/ for the
    overlay's configMapGenerator. chaos <keystone|glance> installs the binary
    via a buildinfo-guarded go install, generates a mode-600 clouds.yaml from
    the ControlPlane admin Secret (missing-Secret and empty-password aborts kept
    distinct), resolves the Keystone auth URL from a docker port probe with a
    :443 fallback, warns without aborting when OTLP ingest looks down, pins the
    OTLP export interval to 15s, and propagates dizzy's own exit code. Both
    subcommands share an atomically-renamed tarball cache under _output/dizzy/.

  4. ee1b0331 feat(kind): add opt-in dizzy VictoriaMetrics + Grafana overlay
    The self-contained deploy/kind/dizzy/ overlay
    (LoadRestrictionsRootOnly, no parent-dir refs). Declares two
    HelmRepositories into flux-system from within the overlay, leaving
    deploy/flux-system/** untouched. VictoriaMetrics-single ingests OTLP with
    Prometheus-naming + promote-all-resource-attributes extraArgs, on NodePort
    30428, emptyDir storage, 30d retention (kind throwaway posture).
    Anonymous-Viewer Grafana with the victoriametrics datasource and a
    dashboards ConfigMap. The repo's first static HTTPRoute attaches Grafana to
    the https-dizzy listener at https://dizzy.127-0-0-1.nip.io. Chart values
    were verified with helm template/show values against
    victoria-metrics-single 0.43.0 and grafana 10.5.15; the only correction was
    moving fullnameOverride to the top level so the server Service renders as
    dizzy-victoria-metrics-server, matching the Grafana datasource URL. Chart
    versions use floating ranges, matching the chaos-mesh/prometheus overlay
    precedent.

  5. 9f583403 feat(hack): gate the dizzy stack behind WITH_DIZZY in deploy-infra
    Adds the WITH_DIZZY flag (default false), cloned from the
    WITH_PROMETHEUS pattern: doc comment, startup-banner line, a gated Step-3
    block that runs hack/dizzy.sh stage-dashboards immediately before
    kubectl apply -k deploy/kind/dizzy, a docker port probe warning when the
    cluster predates the 8428→30428 mapping (deploy continues), a Phase 3 append
    of dizzy-victoria-metrics + dizzy-grafana to the HelmRelease wait list,
    and a post-wait log printing the Grafana URL (honoring KIND_HOST_PORT).

  6. 4edd9c3c feat(make): add dizzy-keystone and dizzy-glance chaos targets
    Two thin wrappers over hack/dizzy.sh chaos <service>. Each runs three
    separated preflights — kubectl reachability, dizzy stack presence, and the
    ControlPlane admin Secret — so failure modes stay distinguishable. The
    version pin stays in hack/dizzy.sh only, so Renovate has exactly one string
    to bump.

  7. 88db7eef chore(renovate): track the DIZZY_VERSION pin in hack/dizzy.sh
    A regex customManager over the single DIZZY_VERSION line, resolving from
    the github-releases datasource for B42Labs/dizzy. Major bumps disabled;
    minor/patch automerge after the standard 3-day cooldown. A regression test
    perl-evaluates the matchString regex against the real script so any drift
    fails fast.

  8. b3f88678 test: pin WITH_DIZZY gating, overlay render, and default-off posture
    Three tests. The flag test locks the three strict WITH_DIZZY=="true" gates,
    the banner, the staged-apply ordering, the Phase 3 append, and byte-for-byte
    survival of the 8428→30428 mapping under a KIND_HOST_PORT override. The
    overlay test pins the SPDX headers on all seven tracked files, the exact
    rendered resource set, and the fail-when-unstaged staging contract (staging
    developer dashboards aside and restoring via an EXIT trap). The chainsaw
    suite asserts the dizzy Namespace, sources, and releases are absent on a
    default-flags cluster, mirroring no-metrics-server-when-disabled.

  9. 06253d11 docs: document the WITH_DIZZY stack and dizzy chaos targets
    Extended quick-start gets an opt-in tip and two architecture-snapshot pod
    rows; the infrastructure-manifests reference documents the overlay (seven
    tracked files, the dashboard staging contract, the always-present listener +
    Certificate vs. the gated HTTPRoute, the 8428→30428 mapping); a new
    dizzy-chaos-testing reference page covers the make runners, environment
    overrides, dashboards, exported metric families, and two troubleshooting
    warnings, wired into the Testing sidebar.

Alignment with the issue

The three decisions settled during elaboration are all honored: host→VictoriaMetrics
OTLP ingest goes through a kind extraPortMapping (not a port-forward or gateway
route); VictoriaMetrics and Grafana install as Flux HelmReleases from the official
charts; dashboards are fetched from the pinned dizzy release at deploy/run time and
staged into a git-ignored directory (not vendored). No divergences from the issue
were observed in the commits.

Verification

Automated coverage lands in commit 8: the deploy-flag test, the overlay-render
test, and the no-dizzy-when-disabled chainsaw suite, plus the Renovate
customManager regression test in commit 7. The overlay chart values were verified
with helm template/show values (noted in commit 4).


Implemented by planwerk-agent c8ef501 with Claude:claude-opus-4-8

berendt added 9 commits July 22, 2026 20:38
Add a second, unconditional extraPortMapping on the kind control-plane
node that forwards host 127.0.0.1:8428 to container NodePort 30428.
This is the host-to-cluster path the upcoming opt-in dizzy
observability stack needs: dizzy runs on the developer host and exports
per-operation OTLP metrics to
http://localhost:8428/opentelemetry/v1/metrics, which the overlay's
VictoriaMetrics server ingests natively.

Listening on 127.0.0.1 keeps the port loopback-only. The mapping is left
in place regardless of the opt-in flag so WITH_DIZZY=true can be enabled
later on any cluster created after this change without recreating the
cluster.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add an unconditional fourth HTTPS listener (`https-dizzy`) on the shared
port 443 of the kind Gateway, routed by SNI to `dizzy.127-0-0-1.nip.io`
for the opt-in dizzy observability overlay's Grafana UI. This is the
repo's first listener that admits routes from another namespace, keyed
by the automatic `kubernetes.io/metadata.name` label, so the gated
overlay's static `dizzy-grafana` HTTPRoute attaches from the `dizzy`
namespace without introducing a ReferenceGrant. The route ships only in
the gated overlay, so with WITH_DIZZY unset no route attaches and the
SNI answers 404, mirroring how the glance listener exists without a
Glance service. A per-hostname self-signed Certificate mirrors the
keystone/horizon/glance siblings and terminates the `https-dizzy`
listener with its own `dizzy-nip-io-tls` key material.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add hack/dizzy.sh, an opt-in wrapper around the dizzy load/chaos
tester with two subcommands. The dizzy version is pinned exactly
once via DIZZY_VERSION so a Renovate regex can bump it. Both
subcommands share a tarball cache under _output/dizzy/ that fetches,
extracts, and atomically renames the release tree into place, so an
aborted download never leaves a half-populated cache.

stage-dashboards copies dizzy's three Grafana dashboards into
deploy/kind/dizzy/dashboards/ for the kind overlay's
configMapGenerator. The chaos subcommand accepts keystone or glance,
installs the binary via a buildinfo-guarded go install, and generates
a mode-600 clouds.yaml from the ControlPlane admin Secret, keeping the
missing-Secret and empty-password aborts distinct. It resolves the
Keystone auth URL from a docker port probe with a :443 fallback,
warns without aborting when the OTLP ingest path looks down, pins the
OTLP metric export interval to 15s, passes DIZZY_ARGS through, and
propagates dizzy's own exit code.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
The dizzy overlay is a self-contained kind addon (kustomize
LoadRestrictionsRootOnly, no parent-directory refs) applied only when
WITH_DIZZY=true make deploy-infra opts in. It declares two
HelmRepositories into flux-system from within the overlay, leaving
deploy/flux-system/** untouched. VictoriaMetrics-single ingests the
dizzy tester's OTLP metrics with Prometheus-naming and
promote-all-resource-attributes extraArgs, exposed on NodePort 30428
with emptyDir storage and a 30d retention window (kind-only throwaway
posture). An anonymous-Viewer Grafana is provisioned with the
victoriametrics datasource and a dashboards ConfigMap wrapping the
three staged dizzy dashboards. The repo's first static HTTPRoute
attaches the Grafana Service to the https-dizzy Gateway listener at
https://dizzy.127-0-0-1.nip.io. The git-ignored dashboards/ staging
directory is populated by hack/dizzy.sh stage-dashboards before
kustomize build resolves the configMapGenerator files. Chart values
were verified with helm template/show values against
victoria-metrics-single 0.43.0 and grafana 10.5.15; the only
correction was moving fullnameOverride to the top level so the server
Service renders as dizzy-victoria-metrics-server, matching the Grafana
datasource URL.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add a WITH_DIZZY opt-in flag to deploy-infra.sh, cloned from the
WITH_PROMETHEUS pattern, so the kind Quick Start stays minimal by
default and only installs the dizzy VictoriaMetrics + Grafana overlay
when explicitly requested. The flag defaults to false and surfaces in
the startup banner next to its siblings.

When enabled, Step 3 stages the pinned dizzy Grafana dashboard JSONs via
hack/dizzy.sh stage-dashboards immediately before applying the
self-contained deploy/kind/dizzy overlay, then probes the control-plane
node for the host 8428 to NodePort 30428 OTLP ingest mapping and warns
without aborting when the cluster predates it. Phase 3 appends
dizzy-victoria-metrics and dizzy-grafana to the HelmRelease wait list,
and a post-wait log prints the Grafana URL, honoring a KIND_HOST_PORT
override.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add two thin wrappers over `hack/dizzy.sh chaos <service>` that run the
dizzy chaos churn soak against the quick-start ControlPlane. Each target
runs three separated preflights — kubectl reachability, dizzy stack
presence, and ControlPlane admin Secret — so the failure modes stay
distinguishable instead of collapsing into one opaque error. The
preflights honor the same DIZZY_SECRET/DIZZY_CP_NAMESPACE overrides as
the script, and the version pin stays in hack/dizzy.sh only so Renovate
has exactly one string to bump.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add a regex customManager over the single DIZZY_VERSION pin line so
Renovate keeps it current, resolving updates from the github-releases
datasource for B42Labs/dizzy. Major bumps stay disabled while minor
and patch updates automerge after the standard 3-day cooldown, matching
the other hack-script pins. A regression test perl-evaluates the
matchString regex against the real script and compares the capture with
the literal pin, so any regex/script drift fails fast.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add three tests that pin the opt-in dizzy stack (VictoriaMetrics +
Grafana) so its default-off posture cannot regress silently.

The flag test locks the default-off gate count (three strict
WITH_DIZZY=="true" gates), the configuration banner, the staged-apply
ordering (dizzy.sh stage-dashboards before kubectl apply -k, under the
same gate), the Phase 3 helm_releases append, and the byte-for-byte
survival of the 8428 -> 30428 OTLP ingest mapping under a KIND_HOST_PORT
override.

The overlay test pins the SPDX headers on all seven tracked files, the
exact rendered resource set (one Namespace, two HelmRepositories, two
HelmReleases, one HTTPRoute, one dashboards ConfigMap), and the
fail-when-unstaged staging contract, while moving any developer-staged
dashboards aside and restoring them via an EXIT trap.

The chainsaw suite asserts the dizzy Namespace, sources, and releases
are absent on a default-flags cluster, mirroring the
no-metrics-server-when-disabled suite.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add an opt-in tip and two architecture-snapshot pod rows to the
extended quick start covering WITH_DIZZY=true make deploy-infra.
Document the deploy/kind/dizzy/ overlay in the infrastructure-manifests
reference: the seven tracked files, the dashboard staging contract, the
always-present https-dizzy listener and Certificate versus the
overlay-gated HTTPRoute, and the 8428 to 30428 host port mapping. Add a
dizzy-chaos-testing reference page for the make dizzy-keystone and make
dizzy-glance runners, covering the environment overrides, the Grafana
dashboards, the exported metric families, and the two troubleshooting
warnings. Wire the new page into the Testing sidebar group.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
@berendt
berendt marked this pull request as ready for review July 22, 2026 21:15
@berendt
berendt merged commit 3803fdc into main Jul 23, 2026
53 checks passed
@berendt
berendt deleted the implement/issue-717-dizzy-stack branch July 23, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate dizzy: opt-in VictoriaMetrics + Grafana stack and chaos-test targets for Keystone and Glance

1 participant